@tomasdavidorg: This pull request references OCPERT-331 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.21-stage-testing-e2e-aws-ipi

@tomasdavidorg: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.21-stage-testing-e2e-aws-ipi

@tomasdavidorg: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

/cc @liangxia @jianlinliu

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tomasdavidorg
Once this PR has been reviewed and has the lgtm label, please assign jianlinliu for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.21-stage-testing-e2e-aws-ipi

@tomasdavidorg: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

The rehearsal job is failing with:

level=fatal msg=failed to fetch Common Manifests: failed to fetch dependency of "Common Manifests": failed to generate asset "DNS Config": getting public zone for "origin-ci-int-aws.dev.rhcloud.com": no public route53 zone found matching name "origin-ci-int-aws.dev.rhcloud.com"

Root cause: BASE_DOMAIN is missing from the env section of all 11 stage-testing configs. Without it, the installer falls back to the default domain origin-ci-int-aws.dev.rhcloud.com, which has no Route53 public zone in the QE/sustaining AWS accounts.

Fix: Add BASE_DOMAIN to each config's steps.env, matching the value used by the automated-release jobs for the same cluster profile:

For example, 4.12–4.16 configs:

tests:
- as: e2e-aws-ipi
  cron: 1 * 31 2 *
  steps:
    cluster_profile: aws-sustaining-autorelease-412
    env:
      BASE_DOMAIN: sustaining-aws-412.devcluster.openshift.com
    workflow: cucushift-installer-rehearse-aws-ipi-stage-testing

And 4.17–4.22 configs:

tests:
- as: e2e-aws-ipi
  cron: 1 * 31 2 *
  steps:
    cluster_profile: aws-autorelease-qe
    env:
      BASE_DOMAIN: qe.devcluster.openshift.com
    workflow: cucushift-installer-rehearse-aws-ipi-stage-testing

Please update all 11 configs and run make update to regenerate the job files.

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.21-stage-testing-e2e-aws-ipi

@tomasdavidorg: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Issue: Missing Stage CatalogSource Setup

The workflow cucushift-installer-rehearse-aws-ipi-stage-testing is missing the stage catalogsource setup step in the pre phase. This is required to configure the cluster to pull operator content from the stage registry before tests run — equivalent to what the Jenkins/Flexy job does via use_stage_index_catalogsource.sh.

Note: The existing enable-stage-catalogsource step in the registry should NOT be used here — it is outdated, uses deprecated ImageContentSourcePolicy unconditionally, and pulls from registry.stage.redhat.io/redhat/redhat-operator-index (requiring stage registry credentials). It does not match the current Flexy approach.

Proposed fix: create a new step enable-stage-fbc-catalogsource

Credentials needed (ref.yaml):

• deploy-konflux-operator-art-image-share in test-credentials namespace (same secret already used by enable-qe-catalogsource and deploy-konflux-operator steps) — provides quay.io/openshift-art pull auth

Commands script logic:

1. Update cluster pull-secret — merge quay.io/openshift-art credentials from the mounted secret into the cluster global pull-secret (openshift-config/pull-secret). Wait for MCP rollout to complete.

2. Create mirror policy — version-gated by OCP version:

   • OCP 4.12 (kube_minor ≤ 26): Create ImageContentSourcePolicy mirroring registry.redhat.io → registry.stage.redhat.io
   • OCP 4.13+ (kube_minor ≥ 27): Create ImageDigestMirrorSet instead (IDMS is the modern replacement for deprecated ICSP)

3. Check marketplace namespace — create openshift-marketplace namespace if absent (OLM has been optional since 4.11)

4. Create CatalogSource using quay.io/openshift-art/stage-fbc-fragments:ocp-<X.Y> — version-gated:

   • OCP < 4.15 (kube_minor ≤ 27): Plain grpc CatalogSource (no extractContent)
   • OCP 4.15+ (kube_minor > 27): FBC format with grpcPodConfig.extractContent (cacheDir + catalogDir)

5. Wait for CatalogSource READY — poll with timeout and emit debug output on failure

Updated workflow

pre:
- chain: cucushift-installer-rehearse-aws-ipi-provision
- ref: enable-stage-fbc-catalogsource    # add this
test:
- ref: openshift-extended-test
- ref: openshift-e2e-test-qe-report
post:
- chain: cucushift-installer-rehearse-aws-ipi-deprovision

@tomasdavidorg It is great we are migrating the job from jenkins to prow. Have we thought of extending it to chat bot so that we can run this job at will?

Hi @asood-rh, it will be part of the job tool https://github.com/openshift/release-tests/tree/main/prow, so you can run the job something like job run_stage_testing --payload quay.io/openshift-release-dev/ocp-release:4.y.z-x86_64.
Also OAR tool an ert-release-bot (Slack) can execute the job by oar -r 4.y.z stage-testing.

Do you mean any other specific chat bot?

Issue: Conflicting CatalogSource Name in Stage Testing Workflow

The new enable-stage-fbc-catalogsource step needs to be added to the pre phase after cucushift-installer-rehearse-aws-ipi-provision, but there is a name conflict that must be handled first.

The cucushift-installer-rehearse-aws-ipi-provision chain already runs enable-qe-catalogsource as its last step, which creates a CatalogSource named qe-app-registry in openshift-marketplace. Since the stage-testing automation code hard-codes the catalogsource name as qe-app-registry, the new stage FBC catalogsource must use the same name.

Fix: The enable-stage-fbc-catalogsource step script must delete the existing qe-app-registry catalogsource before creating the stage one:

# Delete the QE catalogsource created by enable-qe-catalogsource
oc delete catalogsource qe-app-registry -n openshift-marketplace --ignore-not-found=true
oc wait --for=delete catalogsource/qe-app-registry -n openshift-marketplace --timeout=120s || true

# Then create the stage FBC catalogsource with the same name
oc create -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: qe-app-registry
  namespace: openshift-marketplace
...
EOF

This is safe because the deletion happens in pre phase before any tests run, so no Subscription will have been created referencing qe-app-registry yet.

@asood-rh The cluster-bot is not a good fit here. It is designed to provision ephemeral clusters on demand for developer testing/debugging, not to run full e2e workflows against a specific release payload.

Stage-testing is an end-to-end workflow — it provisions a cluster, configures the stage catalog, runs the Stagerun test suite, and tears everything down. The right tools to trigger it on demand are:

• OAR CLI: oar -r 4.y.z stage-testing
• ERT Slack bot: trigger directly from Slack without needing CLI access

Both support specifying the exact release version to test against, which is the key requirement for stage-testing.

@tomasdavidorg presumbit job step-registry-shellcheck is failed due to script issue, I think it was fixed in #76533. can you rebase the code and retry

/test step-registry-shellcheck

Test passed but rather rebasing. Will take a look on the "Missing Stage CatalogSource Setup".

[REHEARSALNOTIFIER]
@tomasdavidorg: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

@tomasdavidorg: all tests passed!

Full PR test history. Your PR dashboard.